LSD Docs/LSD Docs.iso/FILEZ/lsd20.dms/lsd20.adf/Imploder4.doc.pp/Imploder4.doc

Text File | 1990-09-07 | 41KB | 917 lines

                IMPLODER IV
          COMPLETE MANUAL ETC, ALL IN ONE FILE!

Brought to you by LSD!


How to operate the Imploder should be fairly obvious to the average Amiga
user. So I'll try to highlight only those aspects of the user interface
that aren't immediately obvious, and of course this document will explain
how certain selections influence the processing behaviour.


------------------
The User Interface
------------------

** Configuring the Imploder **

When you run the Imploder, either from the CLI or WorkBench, you'll notice
the music and NTSC size screen. This setup is configurable by way of
commandline switches or tooltypes. Note that, under Kick 1.3 or less, when
one sets a tooltype using the WorkBench's info option, the appended "=" is
required for proper recognition.
The default setting is enabled music without modifications to the filter or
NTSC/PAL mode. However, you can disable the music using the "NOMUSIC"
commandline switch, or the "NOMUSIC=" tooltype. The lowpass filter can be
disabled for less muffled sound by specifying "NOFILTER/NOFILTER=".
To people with PAL Amigas, the shrunken NTSC screen might look ugly, however
if there is a fat Agnus chip in your machine, it can be toggled from a 50Hz
to 60Hz display refresh rate. Setting "NTSC/NTSC=" will cause the Imploder to
do this whenever its screen gets upfront.

In addition to this, there's the SHORTROOT configuration switch. This
option is related to using the library. See the "library" document for
further information on this.


** The File Requester **

Whenever there exists the need to specify a filename, a directory window will
appear in front of the message window. Only a few of its options need
clarification:
-Switching on the ID toggle gadget causes the directory window to display
 the type ID's of all files. This is useful, for example, to check if files
 have already been imploded. This option will increase the time needed to
 access a directory though.
-Clicking on the proportional scroll gadget on the left will cause the list
 of filenames to be sorted.
-Switching off the .INFO toggle gadget causes the directory window not to
 display the *.info files used by the WorkBench's icon system.
-Selecting a file is done by placing the mouse pointer on top of its name and
 clicking the left mouse button. In addition to the normal ways of confirming
 your selection, you can double-click on a filename.
-Next to the immediately obvious ways of scrolling through a long directory,
 one can place the pointer within the filename window, press the left button,
 and while holding it down move the mouse above or below the filename window,
 try it; it works great once you get used to it.
-One does not have to wait for the entire directory to load before selecting
 a file or entering a sub directory.


** Keyboard Equivalents **

Many functions selectable by mouse also react to keys. The menu options have
the usual right-amiga+key shortcuts. In addition to this;
- Hitting any key removes the about requester displayed during startup.
- Use the escape key to exit the Imploder.
- The "S" key starts processing.
- When the merge/compression parameter window is present, you can select
  the compression mode using numeric keys, and increase/decrease the merge
  threshold using the "+" and "-" keys. You can cancel using ESC and proceed
  by hitting RETURN.
- The deplode query requester reacts to the "Y" and "N" keys.


** Loading / Saving **

When you have specified a file for the Imploder to load, processing will
commence (detailed below), and once finished the file requester pops up
to query you for the destination.

The source and destination paths are maintained in a special way that
minimizes the chance of the destination not being what one wants. You
could e.g. implode files from one directory and store them in another,
or overwrite them in the source directory.
In the first case it would be annoying if the destination directory defaults
back to the source; you'd have to change it every time you want to write an
imploded file. In the latter case, it'd be annoying to change to a different
source directory and find out that the destination still points somewhere
else.

So the logic the Imploder uses is as follows; if you change to a different
source directory, the destination directory will follow the source directory
only if the destination is equal to the source.


** Batch Mode **

The Imploder can compress a series of executables by allowing you to specify
multiple files by way of ?,#,* wildcards. All non-imploded executables
matching the pattern will be processed. After entering the wildcards, the
"merge and compression" parameter window will appear. The options present
there will be explained below. The important thing to note here is that
these selections will be global to all files processed during batch mode.
Next, you'll be asked to specify a destination directory. This is were all
batched executables will be saved after implosion.


----------
Processing
----------

Now you're familiar with the user interface, let me get on with explaining
how the Imploder processes programs. The implosion processing sequence
consists of four steps;
-Loading & Format Verification.
-Hunk Merging & Reloc Table Cleanup.
-Implosion.
-Decompression code installation, Overlay table adjustment and Saving.


** The Loader **

The loader reads the selected file. During this process, the executable
file is analyzed. Format defects cause the loader to either report an
error, and abort processing, or to give a warning while ignoring or
repairing the defect.

After having loaded a file, the "merge and compression" parameter window
will appear. You can click the topright icon to collapse this window and
review the status information produced by the loader.


** The Merger **

The merger is disabled by default. Most executables produced by modern
compilers/linkers don't require this option, so if you're impatient
you can skip to the next section.

If selected, the merger will try to coalesce the hunks in a program to hunks
of upto the merge threshold in size. If used in a constrained manner, this
will reduce the size of the program, and the chance of memory fragmentation.
If you specify a large merge threshold however, the resulting file will need
large contiguous regions of memory in order to load, thus reducing the chance
of it being able to be loaded into a fragmented system.

Merging an executable file might break it. This is because certain programs
make assumptions about the format of their segment list. The most notable
members of this group are BCPL programs and a few libraries and devices.
Hunk merging is therefore automatically disallowed for these. Still, some
other programs, especially selfdetaching programs might dislike being merged.

Regardless of whether a program has been merged or not, a reloc table
cleanup routine is executed upon the file. This deletes empty reloc tables,
coalesces matching reloc tables, and finally sorts the relocation offsets.
This is of no consequence to how the program will look after it has been
decompressed and relocated.


** Compression **

The compression algorithm can operate either in turbo mode or in normal mode.
The normal mode requires no additional memory, whereas the turbo mode needs
some 300K of additional hashing buffers. The turbo mode kicks-in automatically
whenever the required amount of additional memory can be allocated. It is
about ten to twenty times faster than the normal implosion mode.
Note that the parameter window will report whether or not the current memory
configuration allows the turbo to be enabled or not. You might try to free a
few resources in order to make the target. Whenever you select a compression
mode (gadgets 0-8), the turbo capability will be reevaluated.

Various compression modes can be specified. These modes vary from zero
(range  = 128 bytes) to eight (range = 18K). The range related to each
compression mode determines the maximum distance searched while looking for
redundant data. The higher compression modes therefore have a better chance of
compressing data.

The time required to compress a program increases proportionally to the
maximum distance in case of non-turbo operation. The processing speed during
turbo operation however, is only slightly affected by varying the compression
mode.

So it only makes sense to fiddle with the compression mode if you don't have
enough memory to have the Imploder run in turbo mode, and therefore want
to speed things up (at the expense of a the compression efficiency).

For all other instances it suffices to leave the compression mode at its
maximum value (eight); for small executables the mode will automatically
be scaled down to what the Imploder thinks is probably the most efficient
one for the file at hand.

During compression, the level meters to the right will display various
relevant parameters. Going from left to right these are:

1. Percentage of program data still to be compressed.
2. New size of executable in % of the former size. Reevaluated continuously.
3. Skip. If a large chunk of non-compressable dat is encountered within
   a program, its level will start to rise. When it hits the top, the
   encoding limit has been exceeded. This is a rare occurance.
4. Length. Gives a measure of the redundancy of the data currently being
   processed. High levels indicate good compression.
5. Distance. Gives a measure of the non-locality of current redundancy.


** Decompression Code Installation **

All imploded executables require some additional memory in order to be
able to decompress. The additional amount of memory required is about 50%
of the original program size plus a constant amount of buffer space depending
on the compression mode, and never larger than 20K.
After decompression this memory is freed without causing memory fragmentation.
Furthermore, the normal distribution of program-data across hunks is retained.
This allows for the loading of imploded files into fragmented memory, and
the proper distribution of hunks across chip and fast memory.

The Imploder is able to install 3 different decompression algorithms into
imploded executables;
Library, Normal and Overlayed.


LIBRARY IMPLODED files are the default, they are generated when;
-The "Compress" gadget is enabled.
-The "Library" gadget is enabled.
-The processed executable isn't overlayed.

To be able to use library Imploded files, copy the explode.library to your
LIBS: directory. When you run a library imploded program, the decompression
code in this library will be called. The library code is fast and removes
the need of appending decompression code to every imploded file.

The library has several special useful properties discussed in the "Library"
document, but for simple use it suffices to make sure the library is in LIBS:.

Pure programs may be library imploded.


NORMAL IMPLODED files are generated when;
-The "Compress" gadget is enabled.
-The "Library" gadget is disabled (this is NOT the default).
-The program being processed isn't overlayed.

Normal imploded files have decompression code appended to them. A normal
imploded program will run just like the original, and is independent; in
contrast with library imploded files, normal imploded files don't require
libraries or other special files to be present in your system.

Disadvantages of normal Imploded files are that they are a couple of hundred
bytes larger than library Imploded files, and their decompression speed is
significantly slower. This is due to need for space optimizations in the
decompression code.

Once you normal-implode a pure program it's no longer pure.


OVERLAYED IMPLODED files are generated when;
-The "Compress" gadget is enabled.
-the processed executable is overlayed.

Overlayed programs are executables with additional appended hunks that are
loaded during runtime. The Imploder will automatically recognize whether a
program is overlayed.

Overlayed files are rather loosely defined; basically the program is passed
a bit of information that allows it to get at the appended hunks, but the
means of doing so is left up to the programmer, though there is a standard
technique specified by CBM which is used by the majority of overlayed
executables.

Because the Imploder decreases the size of the file, the offset of the
overlay data also changes, and this must be corrected for. The Imploder
knows how to do this for the standard overlay format. You'll be notified
when the format isn't as the Imploder expects.

Thus imploding overlayed files is not something which is guaranteed to
succeed, so take heed; you should only implode overlayed files when you
know what you are doing, and are willing to verify the results.


---------
Deplosion
---------

If you select an already imploded executable for processing (the ID gadget
in the directory window allows you to see if files have already been imploded,
at the cost of slower listing), you will be asked if you want to deplode it.
If you confirm this query, the executable will be restored to a normal
non-imploded format. Note that this format isn't necessarily bitwise
identical to the original; the reloc tables are sorted, and you might
have applied hunk merging (which is irreversible). Also, any symbol or
debug hunks present in the original will have been stripped.
Still, if you run the program, things will look the same to it, and
this is ofcourse what counts.

Note that there is a CLI based "Deplode" command available in the Tools
directory which does basically the same thing. This is purely a matter of
convenience ment for people that think of the CLI as convenient.




Changes V3.0 -> V4.0
--------------------


Major:

- Finally a complete and up-to-date well documented release with all
  of the support utilities.

- Basic support for batchmode processing.

- Radical new music composition by Paul van der Valk.

- Erwin Zwart has completely redesigned the graphics to conform to the
  2.0 look. Aesthetics and functionality have improved significantly.

- Defaults to support for a "safe library root". This is larger startup code
  for library imploded files that warns when no library is present. You can
  override this setting.

- The "Protect" option has been discontinued. See the "philosophy" document
  for the rationale on this.

- Mouse hater support by way of a CLI based Deplode command, and keyboard
  equivalents throughout.



Minor:

- Support for "pure-imploded" files removed; If you know how to make a
  program resident, you know how to use the library.

- The "Library" gadget now defaults to on.

- The "Merge" gadget now defaults to off. Improved compilers/linkers
  plus the much more frequent appearance of selfdetaching programs have
  made this option less important and more dangerous respectively.

- Added tooltypes and commandline switches for specifying the setup
  options.

- The copperlist has been tuned down a bit, and the screen is now
  dragable.

- The explode.library has been enhanced/sped up a bit.

- Removed bug that caused Intuition to read from location zero. This would
  occasionally crash the machine when location zero was nonzero.
  This bug was already removed in V3.1 (thanks to Arnout).

- All stuff has been enforced and mungwalled so now we've even better
  confirmation that no serious bugs are present.

- Improved overlay file recognition. No longer expects overlayed files to
  have additional space in the hunk table. Searches for the magic longword.

- Decodes ARP resident program tags. Warns against stack gobblers.

- Data hunk ID now preserved upon deplosion. Inconsequential, yet neater.
  Very stringent format checks in the old deplosion routine barf on the
  extra bit required for this, but the old explode library doesn't mind,
  so in that sense it is backwards compatible.

- Lots of other small changes.




** Features **

Library-imploding files is preferable, not only because the result is
smaller, but also because the library is faster, and able to patch the
AmigaDOS executable file loader (LoadSeg), thus decompressing ANY type
of load file as soon as it is loaded into memory, e.g. fonts, devices,
libraries, etc.

If you only library-implode program files, the library will be loaded
when you run a library imploded program for the first time. However if
you wish to implode non-program executables, you'll have to make sure
the library is already resident because these won't load properly until
LoadSeg has been wedged. If you're not yet using the 2.0 OS, there is a
slight complication explained in the section at the end of this document.

To make sure the library is resident, we've provided a small program that
can be run at the head of your startup sequence. It's called "ExpLoad"
and resides in the tools subdirectory. All it does is open the explode
library. Once this is done, the library hooks itself into DOS and remains
resident, even when a system flush occurs.

Note that it suffices to run a library imploded program to make the
library resident, but this is less visible so you might forget about it
when you modify your startup-sequence later on.

Once you've made sure the library is resident, you may merrily go around
your system and library-implode libraries, devices, fonts and even keymaps!

By default the Imploder appends a foolproof piece of startup code to
library imploded files. This checks whether the library is present.
If not it will print an error. If started from the WorkBench a tiny
window will be opened to display the error message.

However this error checking takes up a bit of space, and is redundant
when you make sure the library is resident. For this reason you can
specify the SHORTROOT tooltype/switch. When set, the Imploder will
append only short startup code to library imploded files. This saves
about 250 bytes for every executable.

If library imploded programs with a short root do not find the explode
library they will keep on trying to open it until they succeed. These
programs will therefore hang until the LIBS: directory contains the
library.

Still, using the SHORTROOT switch has its advantages. The reason it is not
enabled by default is to make sure that people have read this document
first.


** Pre-2.0 problems **

If you are NOT using 2.0 or some newer OS, there are a few additional
problems related to BCPL braindeadness, so if you're still running 1.3 or
less, READ THIS!

Under a pre 2.0 OS, if you run a library imploded program generated with
the SHORTROOT option from the WorkBench, while the library hasn't been made
resident yet, the system will guru. This is a problem caused by the OS.
The default library startup code checks for this condition, and will exit
cleanly after reporting an error.

So under 1.3 or less it is advisable to make the explode.library resident
before the WorkBench is active, e.g. early in your startup sequence, even
when you're using library imploded non-program executables;
The first invocation of a library imploded program might happen from the
WorkBench.

In addition to this it is best not to library implode handlers under pre
2.0. See the Techno document for an in depth explanation of these issues.


** Conclusion **

Library implosion limits the possibility of passing on one's executables
to other people/systems who/that do not have the explode.library installed.
I think this doesn't really matter because I strongly believe people
should be left the option of deciding in what way, if at all, they are
going to compress/install executables. (See the "Philosophy" document
for a more extensive discussion of this issue).

Evidently, using library imploded files is the most transparent to the
system. We personally recommend using mostly library implosion, and only
in special cases normal implosion, or overlay implosion for the odd
overlayed file one might encounter.




This doc may sound slightly grumpy in places, still this is how we feel about
things, so there ;-)


Intended use
------------

The Imploder is intended for creating extra space on your system disk
while maintaining full functionality. To achieve this goal, we've tried
to make the decompression process as invisible and fast as possible.

Thus the Imploder doesn't support any annoying colour flashing, and it
has a high speed of decompression. On a vanilla 68000 Amiga, "explosion"
speed is about 30-50 K/s, depending on the type of compressed code. Also,
the explode.library is a bit faster than the explosion routines appended
to stand-alone "imploded" files.

You should take this speed issue into consideration when determining which
executables to implode. For floppy users, the startup times will mostly be
FASTER. Users with fast harddrives however might want to limit themselves
to imploding only infrequently used executables. It is therefore very
useful to floppy users, yet harddrive users can also add a few megs to
their bit budget.

Evidently, the optimum use of the Imploder will vary from system to system.
A3000 owners for example won't even be able notice programs exploding.

Because of this, over the years, the emphasis of the Imploder's intended
use has moved away from Imploding being a once in a program's life time
operation, suitable e.g. when an author wants to distribute his program.
Instead we now feel that every user should be able - when installing a
program - to decide whether or not to implode it, and in what way.


The user's responsibility
-------------------------

So the issue here is freedom of choice. And it is responsibility of the
user to apply the Imploder in a sensible fashion.

To get a feel for what I'm talking about simply look at what happened
when people started distributing Power-Packed text files. Many people
didn't like using the PPMore reader or simply didn't have it. This
kind of thing can be utterly annoying, and the Imploder has the
potential for the same kind of abuse.

We therefore recommend limiting the use of the Imploder to compressing
things installed in your system. So if you must pass on a program to
someone else, use the original archive. This keeps everyone happy,
including the program's author.

Another point is that when receiving programs from PD sources, people
should be able to easily check if programs contain file viri or other
unwanted things. If an executable is imploded it'll have to be deploded
before one is able to examine the code. So don't distribute compressed
executables of _any_ type.

Still, there was one action we could take to encourage the kind of use
we recommend;
In previous versions one could select a "protect" option that prevented
a program from being deplodable. The intent of this option was to protect
an author's work before he distributed it. In reality people started using
it to protect dirty hacks or programs containing file viri.

Evidently this is incompatible with our philosophy. Thus, the support for
protecting programs has been discontinued as of version 4.0. In addition
to this, the Imploder now allows decompression of any protected files
left over from the past.



This document deals with the inner workings. If you're a tech type, and are
interested in somewhat deeper lying aspects relating to how the Imploder
operates, chances are you'll find it here.
Subjects covered are:

- Compression
- Decompression
- Reversibility
- The Library
- Overlayed Files
- Merging
- ARP
- The Music
- The Copperlist
- 68040 Cache Coherency


** Compression **

The Imploder (we only recently learned :-) does LZ77 like compression with a
per-mode static Huffman coding step on the various parts of the skip, offset
and length tuples. Due to the efficient encoding, a tuple can require less
than 12 bits, and thus strings of 2 bytes length are encodable with a decent
gain (given small Huffman patterns corresponding to likely circumstances).

To speed up the string searching, the turbo mode causes the accessible strings
to be indexed using a hashing table. However, the fact that strings with a
minimum size of two bytes are still potential candidates for compression,
requires the hashing function to necessarily be rather simplistic. When the
implosion algorithm processes highly redundant data, entries in the hashing
table tends to get very imbalanced, causing a reduction in performance.
For most types of data, notably executables, this isn't the case though.


** Decompression **

The goal of decompression is to reproduce the segment list of the
original program. This is a linked list of data/code/bss hunks, some
of which require being positioned in chip memory.

The decompression code will have to fill the data and code hunks with
the decompressed data, and, if required, perform relocation of absolute
references.

The Imploder lets LoadSeg allocate all target hunks (as BSS). These
are at the start of the hunk list. This is followed by a hunk with
the decompression code (only for non-library imploded programs),
a compressed data hunk (normally about 50% of the static data size,
depending on compression gain ofcourse), and a decompression buffer
(only upto 17K in size).
As you can see, no allocations need to be done, so the exploding
process will never fail.

During decompression time, data is decompressed from the compressed data
buffer into the decompression buffer until it has filled. It then empties
this buffer by filling hunks and processing reloc tables.
When the buffer has been emptied, the process repeats until all data has
been scatter-decompressed across the target hunks.

Now you might ask, why not directly decompress to the target hunks
instead of using a decompression buffer?
This is because the the Imploder uses the decompressed data to
decompress, and needs to be able to easily access it (for speed).
Referencing data distributed across several hunks is much more
cumbersome.

An added bonus of having separate source and target memory regions is that
a notation can be used that doesn't gain under rare circumstances, but on
average yields better results (No chance of source/target pointer
collision).

When explosion has finished, the decompression buffer, code and data
hunks are freed, and memory usage reduced to what it would have been
for a non-imploded version of the program.

People often compare the Imploder to the Power Packer. The Imploder
decompresses faster and looks cooler [:-)], but the most interesting
differences lie in the implementation of the various steps of the
decompression proccess. So let's contrast the advantages of the
Imploder's approach with the Power-Packer's implementation.

- By having LoadSeg allocate all memory as BBS hunks, explosion will
  never fail.
  The Power-Packer on the other hand allocates hunks. If it fails it
  will simply exit. Power Packed programs launched from the WorkBench
  thus won't reply the startup message, which will be left dangling in
  memory forever.

- Memory doesn't get fragmented. The explosion related hunks are at
  the end of the seglist and thus were allocated (by LoadSeg) AFTER
  the target hunks.
  This isn't true for the Power-Packer. It does leave a hole in your
  free memory list when it frees its decompression stuff.

- Additional memory usage is only about 50% of the static data size +
  the size of the decompression buffer, which is always small relative
  to large programs (maximum 17k).
  So a 30K program might require 62K to decompress (30+15+17), a 300K
  program will require 467K (300+150+17), assuming a 50% compression
  reduction.
  The memory usage report generated after a program has been imploded
  includes BSS hunks. I've discussed only static data here. BSS hunks
  don't require any extra memory usage of course.
  Power-Packed files require a buffer as large as the original program
  for both compressed data storage and decompression. Memory usage is
  therefore always about twice the static data size (again ignoring BSS)
  while for the Imploder it drops to 1.5 for executables large enough to
  matter memory wise.

(In this comparison I'm talking about executables as produced
 by Power-Packer version 3.0b.)

Non-library imploded programs have a small first hunk that calls the
decompression code hunk at the end, and frees these last three hunks.
For library imploded programs this freeing occurs in the library, so no
preceding hunk is needed.


** Reversibility **

Before compression the Imploder preprocesses executables. It kicks out all
the redundant stuff by merging subreloc tables referring to the same hunk,
sorting relocs (improves compression), removing debug symbols etc. etc.
This is what all those info blurbs in the text window are about.

So the deploded executable isn't guaranteed to be byte by byte identical
as far as loadfile control codes are concerned.

What is guaranteed is that the memory images created when the original,
imploded and deploded program versions are loaded are identical.

So the deplosion process isn't 100% reversible. Normally this is no
problem. The reason for uncrunching is mostly wanting to recompress
an executable using a different compression mode, or having a quick
peek at the code e.g. when applying a patch with something like
NewZap.
If however you expect to need the debug symbols, or (important)
require the executable to be in the _exact_ original format in order
to have things like lpatch updates to applied, you're out of luck
if you've only got the compressed executable. So always keep the
original archives/disks!

This is yet another argument for retaining the original archives.
The Imploder is an online space creator not a distribution
archiver (See the "Philosophy" text).


** The Library **

The library code has been unrolled a bit, and optimized here and there
in order to achieve optimal performance. This makes it faster than the
normal explosion algorithm.

If you library implode a program there is NO way in which the program,
after explosion, will be able to notice. If you make sure the library
is resident, this is also true for any executable file loaded for any
purpose by any program.

For normal etc. imploded programs the startup/cleanup hunk mentioned
at the end of the "decompression" section might be detected if a program
goes through contortions involving finding the segment list via murky
DOS structures instead of simply doing PC relative hunk start referencing
which also works from the WorkBench.
I haven't encountered any programs that do this. Still this is yet another
reason to use the library; there is not even the slightest chance of it
being incompatible with an executable.

Note that the Loadseg vector is patched in an "intelligent" manner; it
will install fine for pre 2.0 kickstarts (braindead jumptable format)
as well as in BCPL free systems (2.0+)

Under pre 2.0, when a library imploded file is run from the WorkBench, and
the explode.library isn't resident yet, Exec will try to load the library
from disk. The process's message port however is in use by the WorkBench
reply message, and until it has been replied, it cannot be used by the
DOS in order to send packets. Thus the DOS gurus.

Also, BCPL code doesn't jump through the the library vector. The only
structural problem with this are handlers. These are loaded by the DOS,
and the DOS is BCPL code, again ONLY under < 2.0. Under 2.0 the library
works just like intended when it was first conceived. Transparently that
is.


** Overlayed Files **

The Imploder compresses the load part of an overlayed file as if it were a
normal executable file. Subsequently, the overlay table and the overlayed
program section are appended.
It then tries to adapt the overlay table. Because different types of
overlay supervisors are in use, the format of the overlay table isn't
known to the Imploder. The only assumption made is that the overlay table
contains seek-offset longwords, at longword aligned locations, that point
into the file to the hunk_header ($3F3) identifiers of the overlayed
program sections.
This is how the standard overlay manager operates, but nothing prevents
a programmer with sufficient technical knowledge to create a novel overlay
format (e.g. selfextracting DImp files).

If the Imploder finds one of these offsets, it is adjusted by the amount
the initial load part of the executable file has compressed.
The deplosion algorithm also tries to find these offsets when restoring the
overlay table. Thus there is always a very small chance that the imploder
will adapt a longword that was never meant to be an offset.

An overlayed file gets its information from the loader in four longwords,
at the start of the first hunk. In an imploded overlayed file, this hunk is
the root hunk, and after decrunching these longwords are adjusted and moved
into the first hunk of the actual program (the second hunk of the seglist).

Evidently this process can never be 100% deterministic, so take heed and
test any overlayed programs you've Imploded. Or don't use overlay implosion
at all if you can spare the bits.


** Merging **

Though modern linkers/compilers typically produce executables with one code
hunk and one data hunk, there are still some old executables and less evolved
linkers around. The merging option was implemented when executables with
sufficient hunks to cause a lot of redundancy were still commonplace.

Every hunk requires a longword in the allocation header, plus a hunk ID,
load size, and hunk end ID. That's 16 bytes per hunk, and thus saved for
every merge action. Doesn't sound like much, but generally hunks also
have reloc tables. These waste a lot more space, especially with references
to a lot of different hunks, though there's no easy equation.

The merging step merges matching hunks (data-data, chipdata-chipdata,
code-code) into hunks of upto the merge threshold in size. The actual
size is of course determined by the sum of the sizes of the composite hunks,
and may very well be a bit less than the specified threshold.

Obviously this process discards redundant data in an irreversible fashion,
so deploding the executable won't reverse it.

Lots of tiny hunks cause memory fragmentation, but increase the chance of
the program being able to load when the system is fragmented, and low on
memory. Thus there is a kind of optimal balance that varies from system
to system. In general it can be said that hunks less than 10K or more than
100K are "bad".

Another factor is that loading a program with many tiny hunks causes the
LoadSeg function to issue double as many tiny read commands, thus bogging
down the speed with which an executable can be loaded into memory.

For simplicity's sake, I've chosen for the Imploder to process executables
within a single buffer, without the need for additional backup buffers.
Thus, removing redundant information, and copying hunk data during the
merging and reloc cleanup process involves moving or mirroring large parts
of the buffer. This is why merging can take a while when processing a large
executable with a hundred or so hunks.


** ARP **

Programs written for use with the ARP shell are able to specify the
stacksize they require inside the first hunk of their executable. If
such a program is normal or pure imploded, the segment list won't become
visible until the program is run. Thus ARP has no way of finding out what
the proper stacksize should be.
Library imploded programs have no trouble with this because they are
already decrunched after they are loaded into memory with LoadSeg.
(Provided the library has already been made resident.)

The Imploder will recognize these files and report on them. If the
requested stack-size is larger than the usual minimum (4000 bytes)
a warning will be printed, and you'll be urged to use only library
implosion.

The chance of a programmer relying on a soon to be obsolete shell for
setting a stack LARGER than the usual default is rather slim though.
It would have been very nice if 2.0 had sported such a stack setting
feature, and indeed it had been planned, but was never implemented due
to lack of time on the part of the Commodore programmers.

We'll be on the lookout for any future changes to the executable file
format in order to fix any potential incompatibilities before they'll
cause problems.


** The Music **

When we got word the CIA-A timer was used by the OS under 2.0, we switched to
trying to allocate first CIA-A and if not available CIA-B to drive the music.
However the CIA-B interrupt priority is too high and can interfere with things
like serial transfer. So Paul got this great idea to keep on using a CIA for
precision timing purposes, but drop down to a SoftInt for doing the actual
work, modulations, etc. This works great, the amount of code executed under
CIA priority is now negligible.
Recently, the CATS started feeling guilty about hijacking the CIA-A timer and
thus created "Jumpy the magic timer device". If I understood things correctly
the latest 2.0 timer device moves out of the way and starts using a less
accurate timing source whenever an application tries to allocate the CIA-A.
Pauls music driver can run of both CIA-A and CIA-B, and it would be a pity
to make Jumpy jump without good reason, so he changed the alloction sequence
from A-B to B-A.


** The Copperlist **

There are a couple of unavoidable quirks when one uses copperlists on Intuition
screens. On certain machines, probably PAL, under certain circumstances, dragging
a dense copperlist past scanline 256 or so will cause some video crud to appear
at the top of the display. This can't hurt, but it sure does look ugly. I suspect
this is a hardware misfeature because it ain't fixed yet under 2.0
This was the reason why the screen of older Imploder versions wasn't draggable;
you might just think this muck was our doing.

Second problem is that copperlists "shine through" onto other screens in front.
For this reason we've choosen a colour > 4 for the level bars, so this is never
observable with screens less than 3 bitplanes deep.
The 2.0 OS support proper copperlist clipping, but it has been disabled by
default for compatibility reasons (yeach). Supposedly there is a bit somewhere
in the graphics base to turn this back on, so I'm sure, in due time, there will
be some preference editor to re-enable this.


** 68040 Cache Coherency **

With the advent of the 68040 processor, programs that diddle with code which is
subsequently executed will be prone to some problems. I don't mean the usual
self-modifying code causing the code cached in the data cache to no longer
be as the algorithm expects. This is something the Imploder never had a
problem with, indeed the Imploder has always worked fine with anything
upto and including an 68030.

The reason the 68040 is different is that it has a "copyback" mode. In this
mode (which WILL be used by people because it increases speed dramatically)
writes get cached and aren't guaranteed to be written out to main memory
immediately. Thus 4 subsequent byte writes will require only one longword
main memory write access. Now you might have heard that the 68040 does
bus-snooping. The odd thing is that it doesn't snoop the internal cache
buses!

Thus if you stuff some code into memory and try to execute it, chances are
some of it will still be in the data cache. The code cache won't know about
this and won't be notified when it caches from main memory those locations
which do not yet contain code still to be written out from the data caches.
This problem is amplified by the absolutely huge size of the caches.

So programs that move code, like the explosion algorithms, need to do a
cache flush after being done. As of version 4.0, the appended decompression
algorithms as well as the explode.library flush the cache, but only onder OS
2.0. The reason for this is that only OS 2.0 has calls for cache-flushing.

This is yet another reason not to distribute imploded programs; they might
just cross the path of a proud '40 owner still running under 1.3.

It will be interesting to see how many other applications will run into
trouble once the '40 comes into common use among Amiga owners. The problem
explained above is something that could not have been easily anticipated
by developers. It is known that the startup code shipped with certain
compilers does copy bits of code, so it might very well be a large problem.



The Imploder was conceived, written and performed by:

Albert-Jan Brouwer    -    Programming, documentation.
Peter Struijk        -    More programming, less documentation.
Paul van der Valk    -    Music-programming and composition.
Erwin Zwart        -    Graphics, aesthetic lay-out.


Please direct queries, comments, bug reports, and other things that can
not be resolved by rtfming to:

UUCP hp4nl.nluug.nl!cbmnlux!ecl001!ajbrouw
UUCP cbmvax.commodore.com!cbmehq!cbmnlux!ecl001!ajbrouw
FIDO: 2:281/614 (up for freq sometime fall '91)


Legal mush:

The Imploder is Freely-Distributable, as opposed to Public Domain.
Permission is given to freely distribute this program provided you
include this documentation and other related files, and no fee is
charged in excess of reasonable media and mailing costs.
All programs in this distribution have been enforced and mungwalled.


P.S.

We've been promising a 4.0 version for over a year now, well here it
is. Things got delayed slightly. We apologize for the inconvenience.

End.